Welcome back to deep learning.

And today we want to talk about the final part of the architectures.

And in particular, we want to look into learning architectures.

Also because I'm lazy. So, you know.

OK, part five, learning architectures.

Well, the idea here is that we want to have self-developing network structures

and they can be optimized with respect to accuracy, floating point operations.

And of course, you could simply do that with a grid search.

But typically that's too time consuming.

Recursive self-improvement.

That is really the pinnacle of that, where you then not only learn

how to improve on that problem and on that, but you also improve the way the machine improves.

And you also improve the way it improves itself.

And that was my 1987 diploma thesis, which was all about that.

So there have been a couple of approaches to do that.

And one of the ideas here in reference 22 is using reinforcement learning.

So you train a recurrent neural network to generate model descriptions of networks.

And you train this RNN with reinforcement learning to maximize the expected accuracy.

Of course, there's also many other options.

You can do reinforcement learning for small building blocks transferred to large CNNs,

genetic algorithms, energy based.

And there's actually plenty of ideas that you could follow.

It's pointless to try to tell you what the latest and best version of a, you know,

learn to learn model is.

But they are all very expensive in terms of training time.

And if you want to look into those approaches, you really have to have a large cluster,

because otherwise you aren't able to actually perform the experiments.

So there's actually not too many groups in the world that are able to do such kind of research right now.

A good theory of problem solving under limited resources,

like here in this universe or on this little planet, has to take into account these limited resources.

So you can see that also here many elements that we've seen earlier pop up again.

There's the separable convolutions and many other blocks here in the left hand side.

You see this normal cell with kind of looks like an inception module.

If you look at the right hand side, it kind of looks like later versions of the inception modules

where you have these separations and they are somehow concatenated and also use residual connections.

And this somehow has been determined only by architecture search.

Performance for ImageNet is on par with the squeeze and excitation networks with lower computational costs.

And yeah, there's also, of course, optimization possible for different size, for example, for mobile platforms.

ImageNet, where are we?

Well, we see that the ImageNet classification has dropped now below 5% in most of the submissions.

Substantial and significant improvements are more and more difficult to show on this data set.

And the last official challenge was on CVPR in 2017.

It's now continued on Kaggle.

There is new data sets that is being generated and is needed, for example, 3D scenes,

human level understanding, and those data sets are currently being generated.

There's, for example, things like the MSCoco data set or the Visual Genome data set,

which have replaced ImageNet as state of the art data set.

Of course, there's also different research directions like speed and size of networks for mobile applications.

And in these situations, ImageNet may still be a suitable challenge.

So let's come to some conclusions.